Statistically Sound Exploratory Rule Discovery

Association rule discovery and other exploratory rule discovery techniques explore large search spaces of potential rules to find those that appear interesting by some user-selected criterion of interestingness. Due to the large number of rules considered, they suffer from an extreme risk of type-1 error, finding rules that appear to satisfy the interestingness criteria on the sample data only due to chance. One method for avoiding this risk is to apply a correction for multiple comparisons during the rule discovery process. While this may result in statistically sound rule discovery with tight control over the risk of type-1 error, it introduces extreme risk of type-2 error, rejecting rules that do in fact satisfy the interestingness criteria. This paper proposes a technique to overcome this problem by using holdout data for statistical evaluation. Experiments demonstrate that traditional association rule discovery can result in large numbers of rules that are rejected when subjected to statistical evaluation on holdout data. They also reveal that modification of the rule discovery process to anticipate subsequent statistical evaluation can increase the number of rules that satisfy an interestingness criterion that are accepted by statistical evaluation on holdout data.